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Abstract 

A Delphi method was used to determine the extent to which current educational software 
was meeting the needs of teachers; as well as what changes needed to occur in educational 
software to make it more effective. Five overarching themes emerged: (a) instructional de- 
sign issues, (b) curriculum, (c) materials, (d) cost, and (e) meeting specific needs. The cost of 
software was a concern throughout the study. The belief that educational software should be 
grounded in both content and purpose was also a major concern. Deficiencies and sugges- 
tions for improvement were found. 

OVERVIEW 

In the past two decades, technology has become increasingly prevalent in the 
workings of the educational system, with todays classrooms using more and more 
technology to enhance their curricula (Char, 1990; Heinich, Molenda, Russell, & 
Smaldino, 2002; Jeffries, 2000; Pastor & Kerns, 1997, Perkins, 1995; Shelly, 
Cashman, & Gunter, 2002; Skinner, 2002; Tiu, Guglielmi, & Walton, 2002). 
Nonetheless, how best to utilize and integrate technology effectively into schools 
and classrooms is a question that generates many diverse responses. 

Effective use of technology is a phrase that educators seem to use repeatedly 
when discussing the integration of technology into the classroom (Barrett, 

1993; Newby, Stepich, Lehman, & Russell, 2000), however, it remains unclear 
to what extent educational software really is meeting the needs of teachers and 
students today (Crosier, Cobb, & Wilson, 2002; Cuban, 2001, Forcier, 1999; 
Mills, 2001). 

LITERATURE REVIEW 

Although there is a wealth of information in the current professional litera- 
ture focusing on integrating computer technology into all aspects of the cur- 
riculum, there are two important areas for which there is a dearth of informa- 
tion (Cuban, 2001; Perkins, 1995; Sivin-Kachala, Bialo, & Langford, 1997). At 
the heart of these related areas are software evaluation from the perspective of 
summative evaluation (e.g., student achievement outcomes) and formative 
evaluation of the software (e.g., appropriateness of instructional design elements 
such as content, interface, and degree of computer mediation) and how it is 
used in classrooms (Mills, 2001; Sugar, 2001). This study focused mainly on 
the latter area, that of formative evaluation and instructional design of software. 
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It appears that many commercial educational software publishers do not use the 
time-tested formative evaluation process that is generally accepted by instructional 
designers for most types of educational materials (Boone, Higgins, & Williams, 
1997; Higgins, Boone, & Williams, 2000; Lockard, Abrams, & Many, 1997). 
Without formative evaluation, which is a cornerstone of instructional systems de- 
sign (Dick & Carey, 1990; Fleming & Levie, 1993; Shiratuddin & Landoni, 

2002), the appropriateness of a piece of educational software for a particular stu- 
dent audience is questionable. Additionally, much of the extant research on educa- 
tional software in the classroom has focused predominately on software specifically 
created for research and not software produced in the commercial market (Richey 
& Morrison, 2002; Rosenberg, 1997). This poses a potential problem when ascer- 
taining the value of educational software as curriculum material. 

Teachers rely on experts to produce quality instructional materials for class- 
room use with the assumption that these commercial products have been prop- 
erly designed, developed, and evaluated. However, this is not necessarily the 
case (Shiratuddin & Landoni, 2002; Sugar, 2001). Boone, Higgins, and Will- 
iams (1997) found that commercial educational software publishers are gener- 
ally unwilling to talk when asked about their instructional design process and 
evaluation procedures. Many do not have a set of procedures, and few have 
teachers or students evaluate their software prior to marketing (Higgins, Boone, 
& Williams, 2000; Mills, 2001). 

Even though today’s software tends to be more user friendly than ever, 
many aspects of its design can be very complicated (DiSessa, 2000; 
Hannafin & Hill, 2002; Poole, 1995; Rosenberg, 1997). And although it 
can be argued that many of the traditional materials widely used in the 
classroom may not have undergone a rigorous instructional design process 
(ISD), it can be maintained that it is more critical for educational software 
to undergo a more stringent ISD process than other educational materials. 
This is due to the fact that the educator generally mediates other materials 
as they are being used to make them more effective. In essence, materials 
such as fdmstrips, worksheets, textbooks, and other instructional materials 
go through a formative evaluation process as the teacher interacts with the 
materials and the students. That is to say, the teacher adapts these materials 
to improve and to fit better the needs of the students (Gagne, Briggs, & 
Wagner, 1988; Joyce & Weil, 2000). Often with educational software there 
is less, if any, teacher mediation of the instruction. 

Many teachers lack (a) the expertise to select appropriate software and adapt it 
for use by their students, (b) the technical skills and training needed to evaluate the 
effectiveness of educational computer programs, (c) training in effective pedagogi- 
cal strategies for incorporating software effectively into their teaching, and (d) expe- 
rience and guidance in facilitating computer-based learning within the context of 
time constraints and prerequisite student skills (Drake, 2000; Hinostroza & Mellar, 
2001; Kelley & Ringstaff, 2002; Nations, 2000). Thus, there is a concern as to 
whether the design of educational software does in fact meet basic instructional re- 
quirements for flexibility and attention to individual needs (Hinostroza & Mellar, 
2001; Merrill, 2002; Shiratuddin & Landoni, 2002). 


214 Spring 2004: Volume 36 Number 3 

Copyright © 2004, ISTE (International Society for Technology in Education), 800 . 336 . 5 1 9 / (U.S. & Canada) 
or 54 1 .302.3777 (Inti), lste@iste.org, www.iste.org. All rights reserved. 


Technology integration in the traditional sense referred to courses in com- 
puter programming, keyboarding skills, word processing, or drill-and-practice 
and tutorial software. However, the role of current technology requires educa- 
tors as well as learners to utilize technology as a tool for inquiry, problem solv- 
ing, and collaboration, making it an integral part of learning rather than an iso- 
lated, compartmentalized part of the curriculum (Benson, 2000; Kelley & 
Ringstaff, 2002). Educational software, then, must be designed not only to ac- 
tively engage learners in reflection and inquiry, but must also be cognitively, so- 
cially, and pedagogically appropriate for students (Haugland & Shade, 1994). 
Gardners theory of multiple intelligences (1993) holds that children learn in at 
least seven different ways (i.e., verbal/linguistic, logical-mathematical, visual/ 
spatial, bodily-kinesthetic, musical, interpersonal, and intrapersonal). Designers 
of educational software should bear in mind different learning styles, particu- 
larly when the users are young children (Shiratuddin & Landoni, 2002). 

Although some research has indicated a positive effect for computers in some 
specific educational settings (Elliott & Hall, 1997; Means & Golan, 1998; Roblyer, 
1991; Sandholtz, Ringstaff, & Dwyer, 1997), there remains an absence of sup- 
ported data for much of the application of technology that is used in schools. Ab- 
sent as well from the literature were any data that gave voice to teacher concerns re- 
garding the educational software they were using with their students. 

PURPOSE 

The purpose of this study was to examine the views of technology-using edu- 
cators toward the software that they used with their students. The study devel- 
oped a consensus of what these educators saw as the limitations of educational 
software currently being used and their beliefs about what needed to be done 
for it to be more effective and useful as an integral part of the curriculum. 

DELPHI 

A Delphi method was used to build a consensus in the specific topic area of 
educational software (Hiltz &Turofif, 1993; Sim, 1977). In the Delphi process, 
the participants generated their own opinions and also had the opportunity to 
think about the judgments of others on the topic (Barnette, Danielson, & 
Algozzine, 1978; Hiltz &Turoff 1993). In this process, the individuals partici- 
pated in creating an aggregate opinion and then determined a consensus on the 
topic through a structured series of questions stemming from previously formed 
answers (Hiltz &Turoff 1993; Ricketts, 1985). 

METHOD 

Participants 

The participants included a stratified sample of educational computer special- 
ists (ECSs) and technology-using teachers from 10 elementary schools, 10 
middle schools, and 10 high schools from a large metropolitan school district. 
The ECSs were asked to participate themselves as well as provide two teacher 
participants from their schools based on the following criteria: 
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1. The teacher used educational software at least once a week. 

2. The teacher created assignments that incorporated technology into the cur- 
riculum as opposed to being used simply as playtime when classroom work 
was finished. 

3. The teacher used a computer for his/her own work. 

Of those who initially agreed to participate, 2 1 were educational computing 
specialists (ECS) and 37 were teachers in either a classroom or a computer lab 
setting. 

Research Questions 

Although much discussion has occurred in forecasting for technology needs in 
the near future, much of that discussion has centered on hardware needs and 
connectivity issues for Internet use (Cuban, 2001; Poole, 1995; Roblyer, 
Edwards, & Havrikluk, 1997; Rosenberg, 1997). Very little evaluation or criti- 
cal discussion of commercial educational software has been reported 
(Forcier, 1999; Higgins, Boone & Williams, 2000; Sugar, 2001). The Delphi 
process was used to determine how the current body of educational software 
was viewed by teachers and school district technology experts. Questions that 
were investigated included: 

1 .What deficits do computer-using teachers find existing in current educational 
software? 

2. What adaptations do computer-using teachers routinely make to use educa- 
tional software effectively? 

3. What suggestions do computer-using teachers have for improving current 
educational software? 

4. What changes need to occur in educational software design to meet the needs 
of todays classrooms? 

5. How do computer-using teachers envision the future of educational software? 


Setting 

This study took place in a large metropolitan school district in the southwest- 
ern United States. The district had implemented a technology support system 
in the form of a cadre of educational computer specialists (ECSs). 

Data Collection 

The Delphi process began with the following question sent to each partici- 
pant in Phase 1 of the study. 

Please provide five (5) specific suggestions for improvement and five 
(5) significant deficits associated with the educational software you are 
currently using or have used in the past with your students. You may 
include adaptations that you have made in using the software for it to 
work well in your classroom. 
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A feedback report including a comprehensive list of responses was constructed, 
with similar responses combined and listed only once. The report also included a 
summarization of the seven most frequent items from the original response set. 

In Phase 2, participants were given a survey containing the aggregate list of re- 
sponses from Phase 1 and the summary of the seven most frequent items. They 
were asked to perform three tasks: (a) rate each of the items in importance on a 
five-unit Likert scale (b) select the five most important items from the list, and 
(c) provide a brief explanation for choosing each of the top five. 

Data Analysis 

Domain analysis (Spradley, 1980) was used as the qualitative method to de- 
termine the themes and categories from the Phase 1 Delphi query. Data were 
described using frequency counts, mean scores, and standard deviations. Fre- 
quency scores were calculated in both Phase 1 and Phase 2 of the study. 

Participant counts. Participant counts reflected the number of responses from 
individual participants that fit into particular categories. In this analysis, a par- 
ticipant could have multiple response items in a single category, but the partici- 
pant frequency count for that category would be 1 . 

Top five items. Frequency scores were tallied on all of the items chosen by the 
participants as their top five choices. This information was used in the descrip- 
tion of consensus material. In addition, the frequency scores of the categories 
and themes were tallied. 

Narrative rationales. The narrative rationales linked to the participants’ top 
five choices were examined to see if any additional information was given. In- 
formation beyond the reiteration of the survey items was reported. 

Likert scale data. The Likert scale information was analyzed using both mean 
scores and standard deviations. A mean score was calculated for each item to de- 
scribe the importance of that particular item. 

Consensus. Varying levels of consensus on the different categories or themes 
that emerged were expected. Determining consensus was not the same as 
achieving a majority vote. 

Response tables were constructed to help present and describe much of the 
consensus data. 

RESULTS 

Participants 

With 23 schools agreeing to participate, there were 69 participants possible. 
Fifty-eight of those individuals agreed to participate, giving a participation rate 
of 84% from the total possible participants. Forty-eight participants returned 
the Phase 2 surveys, giving a return rate of 69% of the surveys that went out. 

Phase 1 Results 

After the responses were separated and coded into separate themes and their 
smaller categories, all similar items were easily identifiable. An aggregated item 
was then created to represent these similar items, which reduced the list from 
297 to 78 distinct items. 
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Item frequencies. Responses were grouped into categories and then the catego- 
ries were grouped into themes. Frequencies were calculated for items, categories, 
and themes based on the number of separate responses they represented. 

Items were ranked based on their frequencies. The item frequency was calcu- 
lated on the number of responses the aggregated items represented. For ex- 
ample, the following responses were combined: 

1. Frequently manuals are poorly written. 

2. In general, documentation is either insufficient or tedious. 

3. Better documentation would allow students to quickly navigate the software 
so more time could be spent with the content or purpose of the software. 

An aggregate item, “Manuals and help materials need to be better written,” 
represented all of these items and had a frequency count of 3. 

Top six items. Six items were reported because there was a significant break 
between the frequency counts of the sixth-ranking item and the seventh. The 
top six items (i.e., the six aggregate items with the highest frequency counts) 
were as follows: 

1. Software should be simplified in terms of required operating systems, file in- 
terchangeability, color settings, network/stand-alone versions, use of virtual 
memory, etc. 

2. Quality educational software needs to be less expensive for single purchases, 
network versions, and site licenses. 

3. Software should have multiple modification components to adapt to teacher 
and student needs. 

4. Software is not easy enough to use “out of the box” and requires significant 
time to learn and training with suggestions on how to use the software. 

5. Current software is too predictable, repetitive, uninteresting and does not in- 
corporate relevant and real-life situations into the curriculum. 

6. Software’s content should be grounded in education content and purpose. 

Themes. Instructional Design and Curriculum were the top two themes based 
on frequency counts. Theme frequencies were calculated by adding the frequen- 
cies of the categories that fell under them. Eight themes had a frequency of 1 7 
or more. The cut off of eight was due to a significant difference in the frequency 
counts between the eighth and subsequent themes (see Table 1). 

Categories. Adaptability and External Events were the top two categories with 
the highest frequency counts. The frequencies for categories were calculated as 
the sum of the frequency totals of the items that fell under each category. Six 
categories had a frequency count of 10 or more. The cut off of six was due to a 
significant difference in the frequency counts between the sixth and subsequent 
themes. The top five categories all fell within the Instructional Design Issues 
theme (see Table 2). 

Items. The following six items represented responses from at least 1 5% of the 
participants. 
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Table 1: Top Eight Themes with the Highest Frequency Counts 

Theme 


Frequency 

Percentage 

Instructional Desig 

n Issues 

85 

28.62% 

Curriculum 


37 

12.46% 

Materials 


24 

8.08% 

Teacher Training/Teacher Ideas 

22 

7.41% 

Program types 


22 

7.41% 

Usability Design 


19 

6.40% 

Cost 


17 

5.72% 

Meets Needs 


17 

5.72% 

Note. Values indicate the number of responses and the percentage the theme represents 

Table 2: Top Six Categories 

with the Highest Frequency Counts 

Category 

Frequency 

Percentage 

In Theme 

Adaptability 

16 

5.39% 

Instructional Design Issues 

External Events 

13 

4.38% 

Instructional Design Issues 

User Friendly 

12 

4.04% 

Instructional Design Issues 

Expert Input 

II 

3.70% 

Instructional Design Issues 

Accurate Information 

10 

3.37% 

Instructional Design Issues 

Supplementary 

10 

3.37% 

Materials 


Note . Values indicate the number of responses and the percentage the category represents. 


1. “Quality educational software needs to be less expensive for single purchases, 
network versions, and site licenses” represented 24% of the participants. 

2. “Software should be simplified in terms of required operating systems, file in- 
terchangeability, color settings, network/stand-alone versions, use of virtual 
memory, etc.” represented 21% of the participants. 

3. “Software should have multiple modification components to adapt to teacher 
and student needs” represented 19% of the participants. 

4. “Educational Software should be easier to use, self-explanatory, and more in- 
tuitive” represented 17%. 

5.& 6. The items “Software is not easy enough to use ‘out of the box’ and re- 
quires significant time to learn and training with suggestions on how to use 
the software” and “Current software is too predictable, repetitive, uninterest- 
ing and does not incorporate relevant and real-life situations into the curricu- 
lum” each represented 16% of the participants. 

Phase 2 

Mean scores from Likert instrument. Based on data from all participants, 
there were 13 items that had a mean score of 4.0 or better (see Table 3). The 
two items with the highest average rating of importance both concerned Cur- 
riculum issues. Of the items with an average score of four or higher, three fell 
under the theme Curriculum, three under the theme Materials, two under the 
theme Instructional Design, and one each concerned the themes Meets Needs, 
Upgrades, Cost, Preview/Demos, and One-Computer Classroom. 

Lowest standard deviations for all participants. The item with the lowest standard de- 
viation for all the participants was “Software needs to contain a variety of skill levels to 
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Table 3: Items With an Average Rating of 4 or Higher 

Standard 


Average 

Deviation 

Survey questions Categories 

Themes 

4.52 

0.752 

57. More educational software needs 
to be created that uses higher order 
thinking skills using inquiry methods 
and open-ended questions. 

Curriculum 

4.30 

0.840 

5 1 . Software’s content should be 
grounded in education content and 
purpose. 

Curriculum 

4.26 

0.743 

28. Software needs to contain a 
variety of skill levels to meet the 
needs of students. 

Meets Needs 

4.24 

0.970 

20. Software upgrades should be 
downward compatible and be provided 
to schools as soon as they are put on 
the market. 

Upgrades 

4.20 

1.014 

12. Quality educational software 
needs to be less expensive for single 
purchases, network versions, and 
site licenses. 

Cost 

4.13 

0.909 

5. More educational software should Online 
be available online. 

Materials 

4.11 

0.948 

6. More examples should be provided Examples 
for use in the classroom and for 
modeling purposes. 

Materials 

4.09 

0.915 

53. Software should test student 
mastery of stated objectives. 

Curriculum 

4.07 

0.854 

2 1 . More fully active demo software 
should be provided for evaluation 
purposes. 

Preview/demos 

4.07 

0.975 

36. Software developers should work Expert 
closely, use and incorporate ideas Input 

and suggestions of educators to 
improve their software. 

Instructional 
Design Issues 

4.07 

1.136 

50. All educational software Accurate 

should be hybrid (cross-platform) Information 

so that they work on both platforms. 

Instructional 
Design Issues 

4.07 

1.124 

60. There needs to be more quality 
software aimed at one-computer 
classrooms. 

One Computer 
Classroom 

4.00 

1.033 

1 . Manuals need to be better written Manuals 
with more trouble shooting tips. 

Materials 


Note . Values indicate the mean scores and standard deviations of the survey items with a mean 
score of 4. 0 or higher. 
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meet the needs of students,” with a standard deviation of .743. The item with the second 
lowest standard deviation was “More educational software needs to be created that uses 
higher order thinking skills using inquiry methods and open-ended questions,” with a 
standard deviation of .752. Third lowest standard deviation (.816) was for the item 
“Supplementary materials need to be improved, more detailed and more accurate for 
specific software that makes using the software smoother.” The item with the fourth low- 
est standard deviation, “There is no benefit in using educational software,” had a devia- 
tion of .8 16. The item with the fifth lowest standard deviation (.839) was for the item 
“Current educational software does not meet everyone’s needs.” 

Top five choices. Participants were asked to select the five most important 
items from the Phase 2 survey. Frequency counts were calculated for all partici- 
pants and again separately for teachers only, ECSs only, elementary school 
teachers only, middle school teachers only, and high school teachers only. 

The item that ranked number one for all participants was the single item un- 
der the theme Cost. When recalculated for the separate groups, this item also 
ranked first for all teachers, elementary school teachers, and high school teach- 
ers (see Table 4). 

Ranking second with participants was the item concerning software being hy- 
brid versions, which was an instructional design issue. This item stated that 
there is a need for software to install on either Macintosh systems or Windows 
systems from the same CD (see Table 5). 


Table 4: Response Table for First Choice Survey Item 


Survey item: 

Quality educational software needs to 
work versions, and site licenses. 

be less expensive for single purchases, net- 

Rank: 

All Participants: 1st 

Elementary School Participants: 1st 

Teachers: 1st 

Middle School Participants: 3rd 

ECS personnel: 2nd 

High School Participants: 1st 

Representative Verbatim Responses: 

Software is too expensive. 

Can’t afford site licenses or network versions of software. 


Table 5: Response Table for Second Choice Survey Item 
Survey item: 

All educational software should be hybrid (cross-platform) so that it works on both 
platforms. 

Rank: 

All Participants: 2nd Elementary School Participants: 5th 

Teachers: 12th Middle School Participants: 4th 

ECS personnel: 1st High School Participants: 3rd 

Representative Verbatim Responses: 

More hybrid software for cross platform use 
More crossover between platforms 

Software does not work properly over different platforms and operating systems 
Multiple platform and file format support 
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An item concerning curriculum tied for third. This item pointed out the need 
for open-ended questions and the need for software to use higher-order think- 
ing skills instead of regular drill-and-practice software or information software, 
such as encyclopedias (see Table 6). 

The need for software aimed at the one-computer classroom ranked third as 
well. However, it was ranked first with teachers in general (see Table 7). 

Ranking fifth overall was Internet effectiveness (see Table 8). Participants also 
saw a need for software developers to work closely with other experts such as 
teachers and to incorporate their knowledge in the materials. This item was 
ranked fifth as well (see Table 9). 

Rationales for top five choices. The diverse nature of the responses received as 
rationales precluded the construction of an effective coding system for qualita- 
tive analysis. Some of the responses reiterated the survey item. Some of them 
explained the importance or the item or explained the thinking of the educators 
concerning the effective use of educational software. The rationales that ex- 
plained the choices or demonstrated how the experts were thinking were used 
to discuss and explain the results. 


Table 6: Response Table for Third Choice Survey Item (Tie) 

Survey item: 

More educational software needs to be created that uses higher order thinking skills 
using inquiry methods and open-ended questions 

Rank: 

All Participants: 3rd Elementary School Participants: 6th 

Teachers: 8th Middle School Participants: 4th 

ECS personnel: 4th High School Participants: 3rd 

Representative Verbatim Responses: 

I would like to see software that helps student logically work through the process of 
solving problems 

Use more higher level thinking skills when challenging students 
More flexible or open-ended software 


Table 7: Response Table for Third Choice Survey Item (Tie) 


Survey item: 

There needs to be more 

quality software aimed at one-computer classrooms 

Rank: 


All Participants: 3rd 

Elementary School Participants: 6th 

Teachers: 1st 

Middle School Participants: 2nd 

ECS personnel: l4th 

High School Participants: l4th 


Representative Verbatim Responses: 

Software is intended for each student to have a computer. 

Many times classrooms have access to only one computer. 

Only one computer available 

More English and reading software should be developed, specifically for the one- 
computer classroom. 
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DISCUSSION 
Response Rates 

This study enjoyed an exceptionally high return rate of surveys and other 
query materials in both phases. While Phase 1 had a participation rate of 84%, 
Phase 2 had an effective return rate of 96%. The return rates were very high in 
terms of what is necessary to achieve a reliable and valid data set within a 
Delphi method (Hiltz &Turoff, 1993; Sim, 1977). 

Discussion of Findings 

The study was reasonably successful in detecting consensus about important 
ideas and concerns regarding educational software. Rather than attempting to 
apply an across-the-board rubric for determining overall consensus, however, 
levels of consensus were determined for (a) themes generated from the initial 
Phase 1 query, (b) aggregated items constructed from the Phase 1 results, and 
(c) participant category data from Phase 2. 


Table 8: Response Table for Fifth Choice Survey Item (Tie) 


Survey item: 

Due to current constraints (filters, speed. 

etc.), the Internet cannot be used effectively 

Rank: 


All Participants: 5th 

Elementary School Participants: 6th 

Teachers: 8th 

Middle School Participants: 9th 

ECS personnel: 5th 

High School Participants: 2nd 


Representative Verbatim Responses: 

Hotlinks to Web searches on pertinent information. 

This would provide a vastly greater amount of resource than a Help file with in the 
software. If Internet access is unavailable, the could be disabled. 

Not challenging for upper level learners who thrive on the variety provided by the 
Internet 

Internet has too many firewalls through district intranet system. 


Table 9: Response Table for Fifth Choice Survey Item (Tie) 

Survey item: 

Software developers should work closely, use and incorporate ideas and suggestions 
of educators to improve their software. 

Rank: 

All Participants: 5th Elementary School Participants: 15th 

Teachers: 5th Middle School Participants: 1st 

ECS personnel: 6th High School Participants: 33rd 

Representative Verbatim Responses: 

It appears that developers are not asking the teachers what is useful in the classroom 
for thirty to forty students (applicability). 

Companies do not respond to input from educators about improvements, upgrades, 
and enhancements, etc. 

Work more closely with educators in the design and content of software 

Have teachers help create or create software for areas of education ease of use or user 

friendly for both teacher and student 
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Themes 

Five overarching themes emerged from the study. They included (a) instruc- 
tional design issues, (b) curriculum, (c) materials, (d) cost, and (e) meeting 
specific needs. In order to determine a level of consensus for each of the five 
main themes, the following data were considered: (a) number of responses 
from Phase 1, and (b) rating from “Top Five” selections in Phase 2. 

Instructional design issues. This theme had the overwhelming majority of re- 
sponses from the initial Delphi query in Phase 1 with a total of 85 individual 
response items. These 85 individual responses encompassed 1 1 different catego- 
ries, all dealing with instructional design issues. Thirty-three percent of the 
highest-ranking “Top Five” items that were collected in Phase 2 were items rep- 
resenting the instructional design theme as well. 

Several participants indicated that they considered instructional design issues 
very important. They indicated that it was important that software companies 
work closely with educators and students. Several indicated through the Phase 2 
rationales that they believed that software companies were doing that. This, 
however, appears not to be the case. Rather, software developers rarely follow a 
formal instructional development procedures in regards to evaluation (Boone, 
Higgins, & Williams, 1997; Higgins, Boone, & Williams, 2000; Lockard, 
Abrams, & Many, 1997; Dick & Carey, 1990). With the number of items re- 
lated to instructional design that were generated in Phase 1 and chosen as very 
important by participants in Phase 2, instructional design was considered an is- 
sue that should not be ignored. 

Curriculum. Curriculum was ranked the second highest in importance in Phase 1. 
Likert scale data showed high mean scores for all items associated with curriculum. 

Some of the rationales for items in this theme were statements such as “this is 
just good educational practice.” To educators, these issues were obvious; how- 
ever, educational software developers seem not to have deemed these issues im- 
portant enough (Shade, 1996; Sugar, 2001). 

Materials. Materials included both training materials and supplementary ma- 
terials. The need for better and more accurate materials was expressed by the 
participant group as a whole and by all the subgroups. 

Cost. Overwhelmingly, cost was considered important throughout the study. 
It ranked high in every combination of data, such as frequency counts in Phase 
1 and 2, as well as mean scores in Phase 2. This issue was unexpected, because 
no reference to cost as an educational issue of software was encountered during 
the literature review for this study. 

Meets needs. Meeting needs specifically concerns the need for software to provide more 
teacher options, assessment and monitoring components, and a variety of skill levels to 
meet various smdents’ needs (DiSessa, 2000; Nations, 2000; Newby, Stepich, Lehman, 
& Russell, 2000; McKenzie, 2003). Participants stated that to use technology effectively, 
they needed more control over software to do what the teacher expects it to do. 

Items 

To help determine a level of consensus for specific items, the following data 
were considered: (a) number of similar responses that were aggregated into a 
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single item from Phase 1, and (b) rating from “Top Five” selections in Phase 2. 
Only three items appeared in the list of highest-ranking items for Phase 1 and 
in the list of highest-ranking items from Phase 2. The three general areas of 
consensus were (a) cost of software, (b) ease of use, and (c) educational content 
and purpose. 

The item concerning cost, “Quality educational software needs to be less expen- 
sive for single purchases, network versions, and site licenses,” had the second high- 
est frequency of response in the initial Phase 1 query and was selected the most of- 
ten in the “Top Five” part of Phase 2. The item concerning ease of use, “Software is 
not easy enough to use ‘out of the box’ and requires significant time to learn,” was 
ranked fourth in frequency of response in Phase 1 and was ranked tenth in Phase 2. 
The item concerning content, “Software’s content should be grounded in educa- 
tion content and purpose,” ranked sixth in frequency of response in Phase 1 and 
had the seventh highest selection rate in Phase 2. 

Interestingly, however, four items from Phase 1 that had a frequency of only 
one (i.e., only one participant gave this response) were selected as “Top Five” 
choices by more than 10% of the participants in Phase 2. This indicates that 
the consensus-building process that is attributed to the Delphi method was 
working in this situation. 

Participant Categories 

Different items and different themes were clearly of more or less importance to 
specific subgroups of participants; with a higher level of consensus shown within 
the smaller subgroups. Cost was important to all the groups, but items such as the 
need for software aimed at a one-computer classroom setting were more important 
to middle school participants, who ranked the item second, than to high school 
participants, who rated the item fourteenth. The item “Software’s content should 
be grounded in education content and purpose” was much more important to the 
ECS participants, who rated the item second, while the teachers rated it thirty- 
fourth. The elementary participants ranked the item first, middle school ranked it 
seventh, and high school ranked it thirty-third. This indicates that the different 
subgroups or levels of participants had very different needs. This divergent set of 
views is consistent with the way technology is often viewed within the culture of a 
single school (Matthews, 2002). 

Research Questions 

What deficits do computer-using teachers find existing in current educational sofiware? 

Teacher materials were reported to be deficient in the areas of supplementary 
student materials, teacher training, and lesson integration ideas. Participants in- 
dicated that teacher materials for educational software needed to be improved 
and expanded to be used effectively. Also indicated was a need for better and 
more accurate tutorials for students and for the teachers themselves. 

What adaptations do computer-using teachers routinely make to use educa- 
tional software effectively? 

Participants reported that they adapted their lessons and expectations to the 
software rather than adapting the software to fit the lessons and expectations. 
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This was due to the unavailability of teacher options, which were either too 
limited in scope or nonexistent. Without these adaptive abilities, the instruc- 
tional materials cannot meet the needs of the users. 

What suggestions do computer-using teachers have for improving current edu- 
cational software^ 

Many of the suggestions made by the participants concerned issues of forma- 
tive and summative evaluation for educational software. For example, partici- 
pants believed that “Software developers should work closely, use and incorpo- 
rate ideas and suggestions of educators to improve their software.” Some other 
issues raised included ideas that were not currently in place but would be ben- 
eficial if implemented. For example, one participant suggested that software 
companies maintain a user group on the Internet so that teachers could share 
ideas and lessons. 

What changes need to occur in educational software design to meet the needs 
of today’s classrooms^ 

Participants indicated many instructional design issues for improving educa- 
tional software. These were grouped into three main categories: (a) content, (b) 
interactivity, and (c) usability. Under content, the need for educators to be in- 
volved in the formative and summative evaluation process was mentioned re- 
peatedly. Accuracy of the content was a consistent concern, along with choices 
in media types of content (e.g., graphics, editable text, and high quality sound). 

How do computer-using teachers envision the future of educational software? 

The data did not provide any clear evidence for building a vision of the future 
of educational software. The educators who participated in this research did not 
take a visionary or proactive stand on the future of educational software. 

SUMMARY 

This study uncovered areas in which improvement was needed concerning 
educational software. Various levels of consensus were achieved, with the major 
concerns being cost, curriculum, and instructional design issues. The data pro- 
vided information necessary for the creation of better software and for devising 
necessary support to effectively and efficiently use educational software in the 
classroom. 

Limitations 

One possible limitation to the study was that the information gathered was 
specific to a particular school district. However, the software used in the district 
was indicative of the software used in other school districts nationwide. 
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